The present invention relates to an image sensing apparatus and, more particularly, to an image sensing apparatus capable of increasing quality of an image obtained from a solid-state image sensing device by equipping the image sensing apparatus with a solid-state image sensing device and optical modulation means which has controllable wavelength transmission selectivity characteristics and/or controllable light transmission characteristics.
Variety of image sensing system, such as an electronic still camera, a video camera and a TV conference camera, for obtaining image information by sampling an object by using a solid-state image sensing device have been developed and widely used. A conventional image sensing optical system using a solid-state image sensing device will be described below.
FIG. 44 is a cross-sectional view of an image sensing optical unit using a common solid-state image sensing device used in a video camera and the like, and an optical axis. In FIG. 44, reference numeral 1 denotes an image sensing optical system which includes lenses 1a and an iris diaphragm 1b. Reference numeral 2 denotes a filter for adjusting a quantity of incoming light to a solid-state image sensing device 3 or for cutting high spatial frequency included in the object, and reference numeral 13 denotes the optical axis of the image sensing optical system 1.
Light from the object passes through the lens 1a of the image sensing optical system 1 and the filter 2, then forms an image on a photo-sensing surface of the solid-state image sensing device 3.
Next, how the light from the object formed on the solid-state image sensing device 3 as the image is photoelectric-converted in the solid-state image sensing device 3 will be explained. FIG. 45 is a cross-sectional view of a part of the generally used solid-state image sensing device 3 for color image sensing. In FIG. 45, the same reference numerals as in FIG. 44 denote elements having the same functions, and explanation of these elements are omitted. In
In FIG. 45, reference numeral 4 denotes a color filter formed on the solid-state image sensing device 3; 5, an on-chip lens; 6, light blocking unit; 7, a photoelectric converter; 8, parts each of which corresponds to each pixel of the solid-state image sensing device 3; and 16, an image sensor controller for controlling the solid-state image sensing device 3.
The on-chip lens 5 is attached to the solid-state image sensing device 3, for increasing aperture rate. The color filter 4 is configured so that areas corresponding to respective pixels have different spectral transmittances which transmit red, blue, and green light, for instance, per pixel, and provided in the solid-state image sensing device 3 in order to selectively extract colors of the light from the object.
The light from the object which forms the image on the solid-state image sensing device 3 consisting of a plurality of pixels passes through the on-chip lens 5 and color filter 4, then reaches each pixel of the photoelectric converter 7. Then, the light is photoelectric-converted by the photoelectric converter 7, and stored in a form of electric charge in each pixel. The electric charge stored in each pixel is periodically sent to a charge transfer unit (not shown) under the control of the image sensor controller 16. Thereafter, the image of the object is generated by image generating means (not shown) on the basis of the transferred charge.
A solid-state image sensing device for sensing a color image is shown in FIG. 45. Although it is not shown, there are a solid-state image sensing device for a black-and-white or monochromatic image which is not provided with the color filter 4, and a solid-state image sensing device to which the on-chip lens 5 for increasing aperture rate is not attached.
The solid-state image sensing device 3 as described above is manufactured in such a manner that parts, such as a transfer unit (not shown) and the photoelectric converter 7, are sequentially formed during a semiconductor manufacturing process
Further, the color filter 4 used for color image sensing is formed by a photolithography method or a print method, and the on-chip lens 5 provided for increasing aperture rate is manufactured in a semiconductor manufacturing process, such as photolithography or a dry etching.
As a solid-state image sensing device used in such an image sensing system, there are a line sensor, an area sensor, and the like. These solid-state image sensing devices are used in various fields in accordance with their purposes, and the quality of the solid-state image sensing devices has improved year after year.
Next, spectral transmission characteristics of each area, corresponding to each pixel of the solid-state image sensing device 3, of the color filter 4 which is provided in the solid-state image sensing device 3 will be explained.
FIGS. 46A to 46C and FIGS. 47A to 47D are graphs showing spectral transmission characteristics of the color filter 4 formed on the solid-state image sensing device 3. In these graphs, the horizontal axes show a wavelength (nm) and the vertical axes show spectral transmittance of the color filter 4.
FIGS. 46A to 46C are the graphs showing examples of spectral transmission characteristics of a color filter of primary colors of red, green and blue. Specifically, FIG. 46A is a graph showing the spectral transmission characteristics of a red part of the color filter, FIG. 46B is a graph showing the spectral transmission characteristics of a green part of the color filter, and FIG. 46C is a graph showing the spectral transmission characteristics of a blue part of the color filter.
Referring to the red part of the color filter 4 whose spectral transmission characteristics are shown in FIG. 46A as an example, the spectral transmission characteristics of the color filter 4 will be explained. As shown in FIG. 46A, the red part of the color filter 4 has a high spectral transmittance for light in a long wavelength range areas of the visible light, whereas has a low spectral transmittance for light in other wavelength ranges, thus mainly transmits the light in the long wavelength range. Similarly, each of the green and blue parts of the color filter 4 show high and low spectral transmittances with respect to wavelength ranges, and light which passes through the green and blue color parts is in different wavelength ranges.
Further, FIGS. 47A to 47D are graphs showing examples of spectral transmission characteristics of a color filter of complementary colors of cyan, magenta, yellow and green. Similarly to the spectral transmission characteristics of each color part of the color filter of primary colors as shown in FIGS. 46A to 46C, different color parts of the color filter of complementary colors have different spectral transmission characteristics from each other.
Further, FIGS. 48A and 48B show examples of arrangement of color parts of a color filter of a common solid-state image sensing device used in a video camera and the like. As shown in FIGS. 48A and 48B, the color parts are arranged in a fixed pattern on the solid-state image sensing device.
Further, characters R, G and B in FIG. 4BA respectively show the red, green and blue parts of the color filter of primary colors each having spectral transmission characteristics shown in FIGS. 46A to 46C, and characters Cy, Mg, Ye and G in FIG. 48B respectively show the cyan, magenta, yellow and green parts of the color filter of complementary colors each having spectral transmission characteristics shown in FIGS. 47A to 47D. Each square in FIGS. 48A and 48B corresponds to a pixel of a solid-state image sensing device.
When the aforesaid color filter is used in a solid-state image sensing device, since color parts are placed in a fixed pattern on pixels of the solid state image sensing device, each color of the light from the object is sampled at interval between color parts of the same color. If the distance between neighboring pixels is denoted by p and a single color is concerned, regarding the red parts of the color filter of the primary color shown in FIG. 48A, the red component of light from the object is spatially sampled by 2*p in the horizontal direction of a frame.
Upon obtaining an image by performing spatial sampling by a solid-state image sensing device whose pixels are regularly arranged, the same problem as in the case of discretely sampling a time-continuous signal arises. More specifically, since the distance between neighboring pixels (pitch) is denoted by p, a spatial sampling frequency for an object is given as 1/p. Thus, if the object includes any pattern which causes a signal component having spatial frequency of 1/(2*p) to be generated, then a false signal (noise) is generated, which causes deterioration of image quality.
For example, regarding the red color parts of the color filter shown in FIG. 48A, they are arranged at every other pixels, therefore, a spatial sampling frequency of red light component of an object in the horizontal direction becomes 1/2p. Accordingly, if an object includes a pattern which causes a signal component having spatial frequency of 1/(4*p), a false signal (noise) is generated, which causes deterioration of image quality.
The same or similar problem as described in the previous paragraph arises for the other color parts of the color filter of primary colors or for each color of the color filter of complementary colors, and a false signal (noise) may be generated in relation with a pitch of spatial sampling. Thus, in a conventional image sensing system, such as a video camera, using a solid-state image sensing device, a filter as shown in FIG. 44 is often provided for the purpose of filtering the high spatial frequency components in an object so as to prevent the aforesaid false signal from being generated.
As described above, in a case where an image sensing operation is performed by a solid-state image sensing device attached with a color filter which is composed of color areas each of which is set on each pixel, the solid-state image sensing device has advantages of down-sizing aid simplification of structure of an image sensing apparatus because it is a single unit. However, it also has a disadvantage in that a false signal is often generated and a high quality image is hard to obtain.
In order to solve the aforesaid problem, a method for increasing resolution by using a plurality of solid-state image sensing devices, a method for increasing resolution by changing the relative spatial positions between the incoming light and pixels of a solid-state image sensing device, and so on, are proposed.
FIG. 49 is a schematic view of an image sensing optical system for increasing resolution by using a plurality of solid-state image sensing devices. In FIG. 49, reference numerals 3a, 3b, and 3c are solid-state image sensing devices, and reference numeral 9 denotes a color separation prism.
Light passing through this image sensing optical system is decomposed into different wavelength components of red, green and blue by the color separation prism, and decomposed light of each color component incidents on different solid-state image sensing devices 3a to 3c in accordance with its wavelength. More specifically, the light from an object is decomposed into light in red, green and blue wavelength ranges by the color separation prism, and the decomposed light of different color components incidents on the corresponding solid-state image sensing devices 3a to 3c. Therefore, color filters are not provided to the solid-state image sensing devices in this case.
According to the above image sensing optical system, a solid-state image sensing device 3a, 3b or 3c is provided for the color components and a sampling pitch of a color becomes the same as that between neighboring pixels of the solid-state image sensing device 3. Thus, an image can be obtained in higher resolution in the aforesaid method than an image obtained by using a single solid-state image sensing device 3 shown in FIG. 44. Further, in an image sensing means having the color separation prism 9, solid-state image sensing devices are often arranged so that pixels of the solid state image sensing devices do not align with respect to the direction of incoming light as shown in FIG. 50. This is aimed at realizing further increase in resolution and in image quality by shortening the spatial sampling pitch. An image sensing optical system using the color separation prism as above is often applied to a camera system for broadcasting.
However, according to the aforesaid method of using a plurality of solid-state image sensing devices, although a high quality image can be obtained, there is a problem in that, since a very precise prism is necessary for color separation, the configuration of an image sensing apparatus becomes complicated, and the apparatus becomes large and heavy comparing to an image sensing apparatus adopting a method using a single solid-state image sensing device.
Next, a method for shortening the spatial sampling pitch of a color by using a single solid-state image sensing device will be described with reference to FIG. 51. This method is called xe2x80x9cone chip field sequential sensing methodxe2x80x9d, and FIG. 51 illustrates the principle of the sequential sensing method. In this case, a solid-state image sensing device 15 is not provided with a color filter.
In FIG. 51, reference numeral 18 denotes a rotation filter for selecting a wavelength range of incoming light to be transmitted toward the solid-state image sensing device 15; 19, a filter driver for driving the rotation filter 18; and 20, a driver controller for controlling the filter driver 19.
The filter driver 19 consists of a driving force transmission means 19a for transmitting driving force to the rotation filter 18 and a driving force generator 19b. Further, the rotation filter 18 for selecting a wavelength range of incoming light toward the solid-state image sensing device 15 consists of a plurality of parts, 18a, 18b and 18c, each of which has different spectral transmittance. The part 18a mainly transmits red light component, the part 18b mainly transmits green light component, and the part 18c mainly transmits blue light component.
Further, the rotation filter 18 is coupled to the filter driver 19 via a pulley and rotates about the rotation axis placed at the center. The filter controller 19 is electrically connected to a driver controller 20. Further, both the driver controller 20 and an image sensor controller 16 are electrically connected to an image sensing apparatus controller 26 so that they are coupled to operate.
Next, the operation of the sequential sensing method will be described. First, the rotation filter 18 is set so that incoming light toward the solid-state image sensing device 15 passes through the part 18a which selectively transmits light in the red wavelength range, then an image is sensed by the solid-state image sensing device 15 under this condition. After sensing the image, the filter driver 19 turns the rotation filter 18 so that the incoming light toward the solid-state image sensing device 15 passes through the part 18b which selectively transmits light in the green wavelength range, then an image is sensed. Similarly, an image is sensed when the part 18c which transmits a blue light component is set. Thereafter, the three images obtained as above are combined by an image combining means (now shown) to obtain a complete image. This series of operation is performed by the image sensing apparatus controller 26, the image sensor controller 16 of the solid-state image sensing device 15, and the driver controller 20 of the rotation filter 18 which are coupled to each other.
In the sequential sensing method as described above, since image sampling is performed in a pixel pitch of the solid-state image sensing device for each color, the spatial sampling interval of color become shorter than the interval when a single solid-state image sensing device attached with a conventional color filter is used. Thus, it is possible to lessen chance of generating false signals comparing to a case where a single solid-state image sensing device attached with a conventional color filter is used. As a result, it is possible to increase resolution and quality of an image.
According to the single device sequential method as described above, only one solid-state image sensing device is used, however, a function for turning a color filter becomes necessary. Therefore, there is a problem in that the configuration of an image sensing apparatus also becomes complicated, and the apparatus becomes large and heavy.
FIGS. 52A, 52B, 53A and 53B are explanatory views for explaining methods of increasing resolution by changing relative positions between incoming light and pixels of the solid-state image sensing device 3.
Particularly, FIGS. 52A and 52B show a configuration of an apparatus which increases resolution of an image by shortening a spatial sampling interval of sensing light from an object by shifting the solid-state image sensing device in a plane which is perpendicular to its optical axis. In. FIGS. 52A and 52B, reference numeral 3 denotes a solid-state image sensing device including a color filter; 21, an image sensor driver for shifting the solid-state image sensing device 3 in the plane perpendicular to the optical axis; and 22, a driver controller for the image sensor driver 21.
The image sensor driver 21 is an actuator consisting of a layer-type piezoelectric element, for example, and electrically connected to the driver controller 22. Further, the image sensor controller 16 of the solid-state image sensing device 3 and the driver controller 22 of the image sensor driver 21 are connected to the image sensing apparatus controller 26, and coupled to operate.
Next, an operation of the aforesaid apparatus will be described. As shown in FIG. 52A, the solid-state image sensing device 3 is set at a predetermined position, and an image is sensed by the solid-state image sensing device 3 under this condition. After sensing the image, the image sensor driver shifts the solid-state image sensing device 3 in a plane which is perpendicular to its optical axis. At this time, the shifted distance is a pixel pitch or its fraction. Then another image is sensed at the shifted position. Note the shift of the solid-state image sensing device 3 is shown in reference to the axis 13 in FIGS. 52A and 52B.
The aforesaid operation is repeated to sense images by the solid-state image sensing device 3 at a plurality of positions, then the obtained plurality of images are combined by an image combining means (not shown), thereby obtaining a complete image. As described above, it is possible to increase resolution by increasing spatial sampling points from which the light from an object is sensed, and a high quality image can be eventually obtained.
Similarly to FIGS. 52A and 52B, FIGS. 53A and 53B are explanatory views for explaining a method of increasing resolution by changing relative positions between incoming light and pixels of the solid-state image sensing device 3. The difference between the method described in FIGS. 53A and 53B and the one described in FIGS. 52A to 52B is that, in the method illustrated in FIGS. 52A and 52B, the solid-state image sensing device 3 is shifted with respect to the incoming light, whereas in the method shown in FIGS. 53A and 53B, the incoming light is shifted with respect to the solid-state image sensing device 3.
In FIGS. 53A and 53B, reference numeral 27 denotes a shifting means for shifting the incoming light; 28, a plane parallel plate; 29, a rotation axis of the plane parallel plate 28; and 14, a plate driver for tilting the plane parallel plate 28 about the rotation axis 29. The plane parallel plate 28 and the plate driver 14 are connected to each other via a pulley, gear, and the like. Further, the plate driver 14 is electrically connected to the driver controller 24. The driver controller 24 and the image sensor controller 16 are electrically connected to the image sensing apparatus controller 26, and coupled to operate.
Next, the method of shifting the incoming light with respect to the solid-state image sensing device 3 will be explained. The state shown in FIG. 53A is that the surface of the plane parallel plate 28 is perpendicular to the optical axis 13. In this state, light which is parallel to the optical axis transmits without being shifted by the plane parallel plate 28.
Then, when the plane parallel plate 28 is tilted by some angle about the rotation axis 29 so as to be in a state shown in FIG. 53B, the light traveling in the direction parallel to the optical axis and passing through the plane parallel plate 28 is shifted as shown in FIG. 53B. The shifted amount is given as a function of the rotation angle, and by changing the rotation angle, corresponding positions between the incoming light and the solid-state image sensing device 3 can be changed.
Next, an operation of sensing images will be described. The incoming light from an object toward the solid-state image sensing device 3 transmits the plane parallel plate 28 which is in a state perpendicular to the optical axis of the solid-state image sensing device 3 or in a state rotated by some angle with respect to the optical axis, then incidents on the solid-state image sensing device 3, and sensed as an image. Successively, the plane parallel plate 28 is turned so that the light from the object incidents at the different positions of the solid-state image sensing device 3, then the light is sensed as an image under that condition. At this time, the plane parallel plate is controlled so that a shifted amount of the direction of the light is a pixel pitch or a fraction of the pixel pitch.
As described above, positions of the incoming light incidenting on the solid-state image sensing device 3 are changed by tilting the plane parallel plate, images are sensed, then the obtained plurality of images in the aforesaid manner is combined by the image combining means (not shown). According to the aforesaid method, the spatial sampling pitch for sensing an image of the object is shortened, thereby increasing resolution of the image.
However, according to the method of changing relative spatial positions of incoming light toward the solid-state image sensing device and the pixels of the solid-state image sensing device, a plane parallel plate and a driver for driving the plane parallel plate or a driver for moving the solid-state image sensing device become necessary, thus there is a problem in that the configuration of an image sensing apparatus adopting this method becomes complicated, and the apparatus become large and heavy.
Next, a general characteristics of a solid-state image sensing device will be described.
FIG. 54 is a graph showing output voltage from pixels of a solid-state image sensing device with respect to illuminance of incoming light. As shown in FIG. 54, the solid-state image sensing device has a characteristic where, when illuminance of incoming light is lower than a certain value, the output voltage increases as the illuminance increases, however, when illuminance exceeds a certain value, the output voltage is saturated. Thus, there is a problem in that the sensitivity range of the solid-state image sensing device for photoelectric conversion is narrow, namely the latitude is narrow.
Further, the charge obtained by photoelectric conversion is temporarily stored in each pixel, and when a quantity of incoming light exceeds a certain value, the charge to be stored in each pixel exceeds the pixel""s capacitance. Consequently, a phenomena where the overflowed charge which could not be stored in the pixel flows to neighboring pixels occurs, which greatly deteriorates image quality.
Because of the aforesaid problem of the solid-state image sensing device, when it is used in a camera, an iris diaphragm, a neutral density (ND) filter, or the like, for controlling a quantity of incoming light to the solid-state image sensing device is necessarily provided, or each pixel of the solid-state image sensing device is equipped with an overflow drain, or the like, for preventing the overflowed charge from being diffused.
However, in a camera using the afore-described solid-state image sensing device, an iris diaphragm or an ND filter which can merely control a total quantity of light which incidents on the whole solid-state image sensing device is provided as a solution for the problem of narrow latitude of the solid-state image sensing device and the problem in that the overflowed charge from each pixel flows to neighboring pixels. Therefore, when a scene including objects whose luminances are very different from each other is to be sensed, quality of the obtained image is often not satisfactory. Following is the detailed explanation of the problem.
FIG. 55 is an example of a scene including parts whose luminance""s are very different. Further, FIGS. 56A and 56B are examples of phenomena when the scene shown in FIG. 55 is sensed by using a conventional solid-state image sensing device.
In FIG. 55, reference numeral 10 denotes the sun having high luminance; 11, a tree; and 12, a man standing in the shade of the tree 11.
In this case, since the quantity of light from the sun 10 is very high, the quantity of incoming light is limited by an iris diaphragm. As a result, enough quantity of light from the man 12 cannot be obtained as shown in FIG. 56A. On the contrary, if the quantity of light is controlled so that the man is properly sensed, then the area having high illuminance, i.e., the sun in this case, is saturated, which causes deterioration of image quality, such as blooming and smear.
Besides the aforesaid problems, there is a problem in that sensitivities of pixels differ from each other due to manufacturing error of a silicon wafer upon manufacturing a solid-state image sensing device, difference in thickness and material of silicon oxide film and the like, and stains and scratches on the surface of cover glass of the solid-state image sensing device.
The present invention has been made in consideration of above situation, and has as its object to provide an image sensing apparatus capable of obtaining a high quality image by using optical modulation means which has controllable wavelength transmission selectivity characteristics and/or controllable light transmission characteristics in a simple configuration without causing increase in size and weight.
Another object of the present invention is to provide an image sensing apparatus using an optical modulation element which has controllable wavelength transmission selectivity characteristics so as to shorten the sampling interval and widen the latitude of the solid-state image sensing device in order to overcome a problem caused by the false signal due to the sampling interval, the problem due to a narrow dynamic range of the solid-state image sensing device, and the problem caused by unevenness in sensitivity features of pixels due to manufacturing processes.
According to the present invention, the foregoing object is attained by providing an image sensing apparatus comprising: a solid-state image sensing element consisting of a plurality of pixels; optical modulation means, consists of at least a single layer of an optical modulation material, whose spectral transmission characteristics can be controlled in order to control incoming light toward the solid-state image sensing element; and control means for controlling the optical modulation means.
Further, another object of the present invention is to provide an image sensing apparatus using an optical modulation element which has controllable light transmission characteristics so as to widen the latitude of the solid-state image sensing device by each pixel or by each predetermined part in order to overcome a problem caused by a narrow dynamic range of the solid-state image sensing device, and the problem caused by unevenness in sensitivity features of pixels due to manufacturing processes.
According to the present invention, the foregoing object is attained by providing an image sensing apparatus comprising: a solid-state image sensing element consisting of a plurality of pixels; optical modulation means, consisting of at least a single layer of an optical modulation material, whose light transmittance can be controlled in order to control incoming light toward the solid-state image sensing element; and control means for controlling the optical modulation means.
Other features and advantages of the present invention will be apparent from the following description taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.